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ABSTRACT 



In 1997, the Software Engineering Laboratory at Pontifical 
Catholic University of Rio de Janeiro (Brazil) implemented the first version 
of AulaNet (TM) a World Wide Web-based educational environment. Some of the 
teaching staff will use this environment in 1998 to offer regular term 
disciplines through the Web. This paper introduces Quest, a tool for quiz 
generator and automatic correction incorporated into AulaNet. The main goal 
of Quest is to serve as the Web-based assessment tool of the AulaNet 
environment. Other goals include lowering quiz costs, providing extensive 
reporting, giving feedback to students, and enhancing cognitive aspects of 
assessment. Features of Quest include a graphic interface for editing 
questions, three question presentation styles (multiple choice, true/false, 
and fill in) , editing facilities, question qualifiers, automatic correction, 
display and storage of results for students, and detailed statistics for 
teachers. The architecture of Quest is based on the Web; it has a 
presentation layer, application layer, Lua objects layer (allowing the 
exchange of data between the application and data layers) , and data layer. 
The AulaNet environment has two distinct modules- -the course creation and 
maintenance module (for teachers) and course attendance module (for 
students) . Three figures present the Quest architecture, quiz-editing 
windows, and student quiz -taking window. ( Author /AEF) 
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Abstract: The Internet has great potential for educational purposes. In 1997, the Software 
Engineering Laboratory at PUC-Rio implemented a first version of AulaNet™, a web-based 
educational environment. This year, some of the teaching staff will be using this 
environment to offer regular term disciplines through the Web. This paper introduces Quest, 
a tool for quiz generation and automatic correction incorporated into AulaNet™. Its purpose 
is to support the educational assessment process through the Web and to grab the results of 
this process. 



Introduction 

With the fast emerging of the Internet, the Web has become an increasingly interactive and dynamic medium 
for sharing information. The Web provides an opportunity to develop new learning experiences for the students 
not possible previously [Alexander, 1995]. It has been increasingly popular to move away from the traditional 
model of education and move towards a more interactive, computerized system [Carbone and Schendzielorz, 
1997]. 

AulaNet™ is a project in development at the Software Engineering Laboratory (LES) to deploy an environment 
for the creation and maintenance of Web-based courses designed for the layman. The objectives of AulaNet™ 
are to adopt the Web as an educational environment; to foster a workable transition from conventional 
classrooms to virtual classrooms, giving the opportunity to reuse the existing educational material; and to 
create knowledge communities [Lucena et. al., 1997]. 

Quest is a quiz generator and assessment tool incorporated into the core package of facilities provided by 
AulaNet™. It would be desirable for an interactive Web-based learning environment to have a Web-based way 
of assessing the students, instead of using a paper-based assessment. 



Assessment 

Assessment is defined as a direct measure of what has been learned as a result of instruction on particular 
subjects [Gagne et. al., 1988]. Assessment for distance learning has not yet been thoroughly researched [Dirks, 
1997]. Some of the principles of the common paper-based assessment made in the conventional classrooms do 
not apply in distance assessment, such as reporting feedback and security. 

However, the Internet shall be used to create environments that facilitate an active and customized process of 
learning, supporting cooperation. In such a cooperative environment, assessment may not only involve 
measuring the students' ability to remember and repeat ideas, but should also make people think about problem 
situations and new solutions to them [Cox and Clark, 1996]. The strategy of Quest is to provide a simple 
multiple choice-like assessment tool that can be used not only to test the students but also to enhance the 
learning process. 

In structuring a quiz it is important to work with a theoretical framework of learning. There are several 
learning objective taxonomies as can be seen in [Imrie, 1995], [Gagne, 1982]. Bloom's Taxonomy of Cognitive 
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Levels [Bloom, 1956] divides students’ abilities into six different levels, which are: knowledge, comprehension, 
application, analysis, synthesis and evaluation. Using Bloom's taxonomy, a teacher can make a deeper 
evaluation on the students' performance, instead of only checking their grades. 



Description of Quest 

The main goal of Quest is to serve as the Web-based assessment tool of the AulaNet™ learning environment. 
Its other goals include: 

1. Lowering quiz costs', the process of creating, correcting and reporting student results in a paper-based 
system, of a large number of students, is very time consuming [Tinoco et. al., 1996]. This whole process 
requires time, money and staff that must be handled and administrated. 

2. Making extensive reporting : Quest records all the information about the answers given by the students to 
make a comprehensive report of results for the teacher. 

3. Giving feedback to students', instead of recording the results just for the teacher, Quest gives automatic 
feedback to the students, so they can have all the information about their performance on the fly, instead of 
asking the teacher. 

4 . Enhancing the cognitive aspects of assessment', while editing a question, the teacher tells which cognitive 
domain (from the six outlined by Bloom) he wants to assess. Thus, writing an evaluation must be planned, 
enriching its educational value. 

There are other assessment tools available, such as QUIZIT [Tinoco et. al., 1996], CADAL Quiz [Carbone and 
Schendzielorz, 1997] and QuizSite [University of Indiana, 1996]. However, they do not provide a graphical 
interface for question editing. These systems require that the teacher learn some tag language to create a quiz. 
Quest provides a graphical interface to make editing questions easier, without demanding any kind of computer 
knowledge being necessary. Other features of the tool include: 

a. Three question presentation styles: Quest is a multiple choice-like assessment tool, but provides three 
question presentation styles for question editing: multiple choice (with up to five answer choices), true/false and 
fill in. Later, it may offer matching and listing. 

b. Editing Facilities: the tool offers quiz editing facilities such as moving the position of the questions, 
removing questions and insert questions between existing ones. This is done automatically due to the graphical 
interface (non-textual) provided by Quest. Quest provides an easy way to reedit created quizzes, with a "save 
as" option, for the teacher not to loose the original one. 

c. Question qualifiers: while editing a question, the teacher should give it some qualifiers, for example, the 
topic of the instructional material it will cover, the cognitive domain level it will test and its value. These 
qualifiers will be useful when displaying the statistics generated by the automatic correction of the quiz, giving 
richness to these statistics. 

d. Automatic correction : the process of quiz correction is done automatically by the tool. The teacher only 
needs to specify the correct answers at the time of editing; thus, giving him more time to interact with the 
students. The correction mechanism stores all the data needed to make the statistical results provided to the 
teacher and the performance information handled just in time to the students. 

e. Results displayed to students: since Quest is a self-assessment tool, it is fair the students need not be 
dependent on the teacher to send them an e-mail to know about their performance. Carbone and Schendzielorz 
[1997] argue that this would increase cheating. Yet, as self-evaluation serves to indicate to students, and not to 
the teacher, if they are doing well in the learning process, it is important to the students to have a feedback. 
Thus, when a student cheats, he is aware that he is just trapping himself. 

f Storage of results for the student: usually, the results of an assessment are stored for the teacher, so he can 

have a way of bringing back at any moment the information about any quiz on the course. Quest also stores the 
information to be displayed at any time to the students. This information includes: grade; total quiz points; 




3 



number of questions in the quiz; number of questions the students answered correctly; number of questions 
answered incorrectly; time elapsed during the quiz resolution; the topics enclosed in the questions of the quiz; 
the topics of the questions the students answered incorrectly and an option for the students to revise the quiz, 
comparing the right answers with those they gave. This last feature was included in the tool to provide a source 
of help material for further studying, as these results are formatted to show the problems the learners faced. 

g. Detailed statistics for the teacher: the teacher can have plenty of information derived from the correction 
of a quiz. The main idea behind the statistics provided to the teacher is that he must have a way of seeing the 
students’ learning development, to trace where they are facing difficulties and to compare the performances of 
the group as a whole, not, one by one, individually. These statistics include: 

• List of grades: this list serves to see the grades of each student with an option to revise the answers given 

by all the students. This is the way it is done in the conventional classroom. 

• Statistics based on topics: a result displayed as a list of topics gives the teacher a way to know the points of 

the instructional material the students are having problems with. The topics are atomic instances of the 

course, and interpreting these statistics can be useful to see if the material delivered by the course fits the 
needs and objectives of the students and those of the course itself. 

• Statistics based on cognitive domain: these statistics show if the students are constructing their knowledge 
correctly. It exposes the cognitive abilities in which the students show more difficulty. After this 
verification, the teacher can reshape the contents of the course to solve these problems, really helping the 
students to: remember an idea; understand it; put it into practice; verily its component parts; use it to build 
another idea and evaluate its usage. 

• Statistics per question: statistics showing the troubles the students had in solving each of the questions in 
the quiz. This can be used to see the average difficulty of the questions, and like a cross-reference of the 
two previous statistics it will show whether the students had problems in understanding a topic, or 
applying it or evaluating the usage of another one, for example. 

• Standard scores: standard scores involve calculating average scores and information about how much these 
vary from high to low — the standard deviation. Average scores should be concluded with their standard 
deviation, and listing of the highest and lowest scores possible as well as the highest and the lowest scores 
actually received. It would end with a sentence like this: "There were 35 students who took the quiz. The 
possible scores ranged from 0 to 100, with the students actually scoring from 32 to 95 (average score: 72, 
standard deviation: 6.1)." 



Architecture and Implementation Issues 

The architecture of Quest is based on the Web, where its entire interface is developed, using CGI programs to 
provide all its functionality. The figure below shows the tool’s architecture. 

The architecture has a presentation layer, which is the user interface. It is basically composed of HTML 
templates that help the teacher to create, edit and maintain all the quizzes. There are also some tips to help the 
teacher during the process. For example, if a teacher wants to create an analysis question, the tool offers him a 
list of verbs that are commonly used to write analysis questions. Another task of the presentation layer is to 
make input validation, i.e., verify if the teacher did not leave a field in blank, if he miswrote a date, etc. This 
feature makes Quest a distributed tool, with part of the processing resident in the client machine, due to 
JavaScript code. 

The application layer is responsible for all the functionality provided by the tool. It is composed of CGI 
programs written in CGILua [Hester et al., 1997]. It uses Lua objects [Ierusalimschy et al., 1996] to perform 
the handling of questions, the generation and display of quizzes, automatic correction and the feedback 
statistics of the results for both students and teachers. 

The Lua objects layer offers all the functionality needed to allow for the exchange of data between the CGI 
programs within the application layer and the data layer. It accesses the tool’s database engine and also allows 
for the writing and maintenance of the HTML quiz files. 
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The data layer is composed of two databases and a file directory. The first database is a Microsoft Access 
database used to store the data related to questions and quizzes. The Lua objects of Quest only access this 
database. The second database refers to the data of the course maintenance environment, which describes the 
courses and the students who will use the tool. The file directory contains all the HTML files of the quizzes 
generated. 



Example of Usage 



The AulaNet™ environment has two distinct modules: the course creation and maintenance module, for the 
teachers and the course attendance module for the students. In the course creation and maintenance module, the 
teacher can configure his course to have the service Exam. Since a teacher configured his course to have this 
service, he will be able to create quizzes using Quest. 
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Figure 2: Quiz edition windows 




5 






a a 



o o £> d 



E 



E 










The main screen of Quest is divided in two parts: the options menu and the main area [Figure 2]. In the options 
menu, is listed the edition facilities provided to the teacher. The main area displays the tracking of the quiz 
under edition. The edition of a question is done in a float window [Figure 2]. When the teacher is done editing 
he finishes the quiz and the tool generates its HTML file. 



Then, a student, in the attendance module, takes the quiz and submits it to automatic correction [Figure 3]. 
Right after correction, he receives feedback showing his performance. The teacher, in his module, gets detailed 
statistics collected in the results of the assessment process. 




Cxntv Ztvift 



m qn«-5sr k\m rs- s^u^.u £ d.-iH 



V*ubluv 1 4 >lJui ; 1 jiur.lvt*. ‘f«|jicus 

WjxL 

Clfadfi 

0,4 

c* 



1 (velar ! Lpcnr.t©?. Itpira: 
:jc: 



Kc yju. s i a: |" 



Figure 3: Student quiz taking window 



Conclusions 

The Internet has great educational potential [Eales and Byrd, 1997] and Quest is a tool for generating quizzes 
and assessing materials on the Web. The process of creating and editing questions is made on a graphical Web 
interface, thus making the tool completely Web designed. It also aims to help teachers assess all the cognitive 
abilities of the students, not only their ability to memorize information. This tool delivers statistics both to the 
students and the teachers so that they can see if the material offered on the course meets all their purposes and 
needs. 
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